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ABSTRACT  -  This  position  paper  describes  a  vision  of  how  Chemical  Biological 
Radiological  and  Nuclear  (CBRN)  situation  awareness  and  threat  assessment  [1],  [2]  can  benefit 
from  the  use  of  Gesture-Directed  Sensor-Information  Fusion  (GDSIE).  It  describes  a  concept  of 
operations  for  the  war  fighter’s  use  of  electronic  wireless-communication  gloves  (eGloves)  to 
communicate,  plan,  and  react  while  wearing  Mission-Oriented  Protective  Posture  (MOPP)  gear. 
It  also  suggests  the  benefit  of  using  the  hardware  and  firmware  resident  on  the  eGloves  to 
perform  data  fusion  from  environmental  sensors.  It  provides  a  roadmap  of  research  issues  and 
challenges  that  will  need  to  be  overcome  to  realize  this  technological  advance. 

INTRODUCTION 

The  battlespace  is  full  of  threats  that  would  be  better  to  detect  earlier  rather  than  later.  War 
fighters  are  in  the  best  position  to  detect  CBRN  threats  in  theatre.  Current  Chemical-protection 
for  war  fighters  on  the  ground  inhibit  electronic  communication  via  keyboards,  cell  phones,  and 
remote-control  devices.  War  fighters  need  better  communication  methods  in  hazardous 
environments  characterized  by  CBRN  agents.  Important  topics  in  wartime  communications 
include  but  are  not  limited  to  situation  and  threat  assessments.  To  improve  communications 
capabilities  for  the  war  fighter  wearing  protective  gear  in  hazardous  environments,  a  series  of 
eGloves  have  been  developed  with  a  view  toward  freeing  the  war  fighter  of  the  need  to  type  on  a 
keyboard  while  wearing  a  MOPP  suit.  (See,  for  example,  [5]  and  [6].)  These  eGloves  can  help 
the  warfighter  transmit  gestures  with  the  hands  and  fingers  from  within  the  protective  gear  [5], 
[6],  or  they  can  be  used  to  transmit  encoded  ASCII  characters  [5].  Fig.  I  shows  an  eGlove 
prototype. 


Fig.  I .  eGlove  with  motion  sensors  and  CPU  circuitry 
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Research  has  demonstrated  that  chemical  and  biological  sensors  can  he  made  smaller,  more 
sensitive,  more  species  specific,  and  easier  to  deploy  in  a  variety  of  ways.  (See,  for  example,  [1] 
and  [3].)  The  current  eGloves  have  magnetic  and  motion  sensors  for  gesture  recognition  [5],  [6]. 
An  important  future  step  to  enhance  the  effectiveness  of  the  war  fighter  is  to  integrate  CBRN  and 
other  sensors  into  the  eGloves.  Development  of  efficient  data-fusion  algorithms  to  fuse  the 
information  from  CBRN  and  other  sensors  with  gestures  from  the  eGlove  operator  will  provide 
SA  beyond  the  readings  from  individual  sensors.  The  following  sensors  could  be  integrated  with 
gestures  transmitted  using  the  eGlove:  CBRN  sensors,  acoustic  sensors.  Geo-positional 
sensors(GPS),  optical  sensors,  physiological  sensors  to  monitor  the  health  of  the  operator,  and 
imagery  sensory  input  from  video  cameras  mounted  in  various  strategic  locations  in  the 
environment.  The  output  of  these  passive  sensors  can  be  transmitted  along  with  the 
magnetic/motion  sensor  output.  This  will  improve  communications  by  providing  explicit 
measurements  or  information  without  burdensome  and  error  prone  manual  relay.  Integrating  the 
environment  sensor  data  with  the  current  eGlove  gesture  software  would  expedite 
communications  by  providing  contextual  information  to  the  glove.  For  example,  if  the 
physiological  sensors  on  the  glove  are  detecting  values  beyond  normal  thresholds,  different 
communications  suggestions  could  be  presented  than  during  normal  operating  conditions. 

DESCRIPTION  AND  BENEFIT 

Gesture-Based  Sensor-Information  Fusion  (GBSIF)  refers  to  the  fusing  of  sensor  data 
collected  from  the  environment  with  data  from  motion  sensors  on  the  eGlove.  The  eGlove 
features  a  CPU  that  is  used  to  fuse  hand  and  finger  motions  and  positions  into  gestures,  as  shown 
in  Fig.  1.  The  same  CPU  can  be  used  to  fuse  additional  data  from  the  environment.  In  GBSIF, 
the  operator  transports  the  sensor  array  but  does  not  take  an  active  role  in  determining  the 
sensors  that  will  participate  in  the  fusion  or  the  target  subjects  about  which  the  data  will  be 
collected,  with  the  exception  of  the  sensors  mounted  on  the  eGlove.  Data  are  collected  from  the 
environment  and  also  from  the  glove  sensors,  and  these  data  can  be  fused  and  integrated  on  a 
network  site  that  differs  from  the  user’s  node.  Thus,  gesture  sensor  data  and  environmental  data 
are  collected,  fused,  and  integrated  where  appropriate.  However,  the  gestures  themselves  are  not 
the  primary  driving  force  in  selecting  information  sources  and  controlling  the  fusion  process. 

In  contrast,  Gesture-Directed  Sensor-Information  Fusion  (GDSIF)  includes  GBSIF  but  extends 
it  to  the  active  participation  of  the  eGlove  operator  to  initiate  sensor-information  fusion.  The 
concept  of  operation  of  the  GDSIF  is  that  the  war  fighter  would  point  to  a  platform  or  another 
object  in  the  battle  space  using  a  gesture  while  wearing  a  GDSIF-equipped  eGlove.  The  eGlove 
would  be  linked  to  reference  sensors  to  determine  orientation  and  azimuth  of  the  operator’s  arm. 
The  eGlove  also  would  use  GPS  to  determine  the  operator’s  geographic  location.  Gestures  would 
cue  sensors  to  send  their  data  to  the  eGlove  where  these  data  would  be  fused  with  the  gesture  that 
prompted  the  data  collection.  Fusion  would  be  accomplished  in  the  CPU  mounted  on  the  eGlove. 
Our  preliminary  experiments  show  that  the  current  CPU  will  support  additional  sensors  without 
difficulty.  Of  the  many  options  for  accomplishing  this,  the  three  subsequent  architectures  provide 
the  best  advantages. 

1.  Tracking  the  date-time  group  of  the  sensor  data,  the  GPS  geographic  location,  and  the 
sensor  type  will  provide  the  pedigree  metadata  [2]  necessary  to  sort  and  fuse  the  information. 
CBRN,  acoustic  and  GPS  sensors  can  be  mounted  on  the  eGlove  and  the  CPU  can  collect  and 
fuse  data  from  all  sources  on  the  eGlove.  The  CBRN,  acoustic  and  GPS  sensors  would  detect 
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environmental  conditions  but  not  health-related  conditions  of  the  operator.  Redundant  sensors 
mounted  on  the  eGlove  could  help  reduce  problems  due  to  false  alarms  of  any  one  sensor.  CBRN 
Sensors  with  heightened  sensitivity  could  transmit  detection  even  before  the  war  fighter  is  aware 
of  the  presence  of  a  chemical  or  biological  agent.  These  networked  and  fused  sensors  could 
provide  warnings  and  alerts  to  be  sent  not  only  to  the  operator  but  also  to  the  wireless  network 
with  which  the  operator  is  in  communications.  When  applicable,  these  alerts  could  be  localized 
to  prevent  ‘noise’  being  sent  to  unaffected  parties.  This  could  be  used  to  alert  and  send  early 
emergency  response  team  independent  of  any  specific  request  from  the  war  fighter. 

2.  Same  as  1,  but  in  addition  to  external  sensors,  the  eGlove  also  would  detect  physiological 
data  of  the  operator,  such  as  pulse  and  Galvanic  skin  response  (GSR).  For  example,  if  the 
chemical  detectors  sent  signals  about  a  plume  and  the  operator  did  not  transmit  any  gestures 
determined  to  have  intent  or  meaning,  this  could  mean  that  the  chemical  agent  had  attacked  and 
incapacitated  the  operator.  The  pulse  and  other  physiological  signals  could  help  determine  the 
operator’s  health. 

3.  The  CPU  mounted  on  the  eGlove  could  be  programmed  to  detect  not  only  local  signals 
from  hand  and  finger  gestures  but  also  from  a  variety  of  wireless  sensors  throughout  the 
battlespace.  This  could  include  sensors  mounted  on  or  deployed  from  UAVs  [3],  motor  vehicles, 
or  even  other  war  fighters.  This  provides  a  wider  coverage  of  the  battle  space  beyond  the 
immediate  vicinity  of  the  operator.  The  obvious  disadvantage  of  these  extra  sensors  is  that  it  adds 
complexity  to  the  task  of  the  CPU  on  the  eGlove.  The  challenge  is  to  develop  effective  data- 
fusion  algorithms  to  manage  the  data  streams  and  the  metadata  from  an  increased  number  and 
variety  of  sensors  in  the  battle  space. 

The  operator  could  select  a  1)  “Raw-data-only  mode”  to  transmit  the  data  to  the  wireless 
network,  or  2)  “fuse  data”  mode  in  which  algorithms  stored  in  the  eGlove’ s  CPU  would  perform 
data  fusion  and  then  transmit  the  fused-data  result  to  the  wireless  network.  The  operator  could 
use  gestures  to  control  multiple  modes  of  operation  of  the  eGlove. 

GESTURES 

Simple  gestures  can  be  used  to  communicate  information  to  improve  situational  awareness, 
send  commands  to  personnel  and  to  robots  [6],  and  send  commands  to  CBRN  and  other  sensors 
in  the  battle  space.  For  example,  the  gesture  to  use  most  often  for  information  fusion  would  be  to 
point  at  a  sensor-data  source  in  the  battle  space  with  the  index  finger  extended  and  the  other 
fingers  touching  the  palm,  (to  distinguish  it  from  similar  gestures  that  use  the  whole  hand  to 
point.)  This  pointing  gesture,  when  recognized,  would  signal  the  sensor  and  trigger  a  data  stream 
or  a  single  reading  from  the  designated  sensor  to  the  local  common-data  backbone.  Sucessful 
transmission  from  the  sensor  would  trigger  haptic  feedback  [5]  on  the  operator’s  glove 
indicating  that  the  data  set  has  been  sent  to  the  network.  Continuing  the  example,  the  war  fighter 
could  repeat  the  pointing  process  with  a  second  sensor  and  then  a  second  gesture,  for  example  a 
fist  with  the  arm  held  straight  down,  would  trigger  a  pre-determined  sensor-information  fusion 
process.  Using  the  fusion-fist  gesture  in  this  manner  would  distinguish  it  other  gestures  that 
employ  a  closed  fist  with  the  arm  extended,  which  in  some  command  contexts  means  “stop.”  It 
also  would  avoid  confusion  with  gestures  in  which  the  fist  is  held  close  to  the  chest. 

CBRN  sensor-information  could  be  thus  queried  and  fused  with  each  other  and  also  with 
information  from  other  sources  in  the  battle  space,  such  as  electromagnetic  or  acoustic  sensors. 
In  more  advanced  implementations  of  this  methodology,  the  war  fighter  could  provide  some 
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degree  of  input  about  what  type  of  information  is  desired  from  the  sensor-information  fusion, 
such  as  a  prediction  of  routes  to  use  for  relatively  safe  travel  with  respect  to  the  deployment  of 
CBRN  agents. 

CHALLENGES 

To  facilitate  GBSIF  and  GDSIF,  the  vocabulary,  syntax,  and  semantics  of  command  gestures 
would  be  developed  to  include  all  possible  modes  of  operation  that  the  war  fighter  would  need. 
Since  the  eGlove  should  be  used  in  a  variety  of  situations,  the  vocabulary  of  gestures  for  GDSIF 
ideally  would  build  on  the  vocabulary  of  gestures  already  in  use  for  similar  purposes.  For 
example,  special  forces  already  have  gesture-based  communication.  Human  factors  and 
physiology  would  help  determine  the  ease  of  using  certain  gestures  rather  than  others.  Special 
fusion  algorithms  would  be  developed  to  process  gesture  data  and  fuse  them  with  environmental 
data.  Advanced  operators  could  be  trained  to  issue  gesture  commands  to  determine  which  fusion 
algorithms  to  execute.  These  are  areas  that  require  future  research. 

Maintaining  the  integrity  and  timeliness  of  the  Common  Operating  Picture  (COP)  with 
existing  data  is  already  a  challenge  [4].  Integrating  these  GDSIF  data  into  the  COP  where  other 
war  fighters  could  benefit  from  local  observations  is  an  even  greater  challenge  requiring  further 
research  and  testing.  Potentials  include  integration  into  the  Joint  Warning  and  Reporting 
Network  (JWARN)  program  of  record. 
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